1. Field of the Invention
This invention relates to multi-processor systems and more particularly to interrupt processing in multi-processor systems.
2. Description of the Related Art
In general, an interrupt (e.g., exception) is an event that changes instruction execution from a currently executing instruction flow to another instruction flow. An interrupt is typically generated by a processor or device coupled to a processor. A typical interrupt processing mechanism changes program control flow of the interrupted processor to an interrupt handler (e.g., interrupt service routine). Referring to FIG. 1, an exemplary multi-processor system (e.g., system 100) includes at least two processor cores (i.e., central processing units, cores, or hardware accelerators) configured to concurrently execute multiple application threads. An exemplary interrupt delivery mechanism (e.g., an interrupt delivery mechanism of the x86 architecture) includes an interrupt controller (e.g., a local Advanced Programmable Interrupt Controller (APIC)) for each core in the system. In addition, an interrupt controller (e.g., I/O APIC) may be included for each peripheral bus in the system. A dedicated bus or a system bus (e.g., crossbar 116) may be used to communicate between APICs.
In general, a local APIC manages external interrupts for a corresponding core or CPU. The local APIC is able to accept and generate inter-processor interrupt (IPI) messages. Exemplary IPIs occur when a first core of a multi-processor system offloads a parallel task to another core. A typical local APIC supports up to 224 usable interrupt vectors, which correspond to interrupt priority and respective interrupt service routines. Another 32 vectors are reserved for interrupt processing by the associated core or CPU.
Referring to FIGS. 1 and 2, an inter-processor interrupt is generated by a core (e.g., core 102) that writes to the Interrupt Control Register (ICR) in a corresponding local APIC (e.g., local APIC 106) (202). An ICR includes fields for a destination identifier, delivery mode, an interrupt vector, and other suitable information. Local APIC 106 generates an interrupt message and broadcasts the interrupt message through the on-chip network using crossbar 116 (204). A local APIC (e.g., local APIC 108) that receives the interrupt message determines whether to accept the interrupt based on a state of the associated core (e.g., core 104) and a state of the receiving local APIC itself (e.g., a delivery mode and a destination identifier). For example, if local APIC 108 has an identifier, physical ID, or logical ID that matches the destination ID according to the delivery mode, the local APIC 108 accepts the interrupt message, reads an interrupt vector number from the interrupt message, and a corresponding bit is set in the Interrupt Request Register (IRR). Local APIC 108 sends an ACK (acknowledgement) message to local APIC 106 that originated the interrupt message to confirm the acceptance of the interrupt message by local APIC 108 (206).
The interrupt vector number read by local APIC 108 from the interrupt message represents a priority of the interrupt, which is compared to a priority of other pending interrupts and a priority of one or more currently executing threads. If the interrupt has the highest priority, a corresponding bit in an In-Service Register (ISR) is set and the interrupt is delivered to core 104. Logic in local APIC 108 sends a message to core 104 (208). Receiving the message, core 104 detects the interrupt and at an instruction boundary of the currently executing application thread, the core executes an interrupt service routine. The interrupt service routine accesses an Interrupt Descriptor Table (IDT) based on contents of an Interrupt Descriptor Table Register (IDTR) and obtains a code segment selector and an offset and privilege mode of an interrupt handler corresponding to the interrupt vector. Once the interrupt handler entry point is determined based on a segment selector and offset program control is transferred to the interrupt handler, core 104 handles the interrupt by executing actions specified in the interrupt handler (212). Control returns from the interrupt handler and may return to a previously executing application thread, according to results of those actions specified by the interrupt handler (214).
Still referring to FIGS. 1 and 2, an interrupt from a device (e.g., device 112) coupled to a peripheral bus (e.g., peripheral bus 114) of a multi-processor system (e.g., system 100) is handled similarly to inter-processor interrupts, as described above. For example, in a network packet processing system, device 112 is a network interface card (NIC). When packets arrive, the NIC sends an interrupt to the processors to notify the processors of the arrival of the packets. Device 112 generates an interrupt by asserting an interrupt signal (202) and the I/O APIC (e.g., I/O APIC 110) reads a corresponding entry in an Interrupt Redirection Table (IRT) 116. I/O APIC 110 generates an interrupt message with information from the entry, e.g., destination identifier, delivery mode, interrupt vector, or other suitable information. Then, the interrupt is broadcast to the local APICs (e.g., local APIC 106 and local APIC 108) (204). A destination local APIC (e.g., local APIC 108) sends an acknowledgement to the I/O APIC (206). Then, the interrupt is delivered to a receiving core corresponding to the destination local APIC (e.g., core 104). Core 108 processes the interrupt (210, 212, 214, 216) in the same way that core 108 processes inter-processor interrupts.
Sending an interrupt from an application executing on one core to another core requires an originating core to transfer control to the operating system because the ICR can be written only in kernel-mode. An application that generates an interrupt to another core switches into kernel-mode to write the ICR and then switches back into user-mode, requiring two control transfers. A typical transfer of control to the operating system uses many cycles (e.g., thousands of cycles to tens of thousands of cycles or more depending on the complexity of an operating system implementation). Moreover, if the interrupt is used to pass information to application threads, an additional transfer of control is required to transfer the program control back to the application thread. Such high overhead of using the interrupt mechanism associated with the operating system for processing user-level interrupts may slow down system performance.